fix(seidb): fix stale FlatKV migration gauges on snapshotting nodes by blindchaser · Pull Request #4436 · sei-protocol/sei-chain

blindchaser · 2026-10-02T19:53:30Z

On atlantic-2, archive-0-0-0, snapshotter-0 and state-sync-node-0 finished the EVM migration (all 123,876,555 keys moved, seidb_migration_version went to 1), but the FlatKV migration dashboard still shows them as migrating, at 99.3% with ~875k EVM keys left in memIAVL. The migration is complete; two gauges report stale values, and only on nodes that export state-sync snapshots. A snapshot export opens a read-only composite store at the snapshot height, and when that height is before completion, the MigrationManager built for that handle records version 0 on the process-wide seidb_migration_version gauge. Nothing records 1 again until restart. Separately, rootmulti.Store.Snapshot records iavl_total_num_keys only for stores that exported at least one node, so once the memIAVL evm tree is empty its last pre-migration value is exported forever. This is the same retention problem #4327 fixed for seidb_migration_boundary_snapshot.

migration.BuildRouter now takes RouterOptions, and WithoutTelemetry() gives the router's MigrationManager newLocalMigrationMetrics() instead of the OTel-backed instance. CompositeCommitStore.buildRouter passes it for derived stores (the LoadVersionReadOnly view and Copy), so only the live store publishes migration metrics. Snapshot sets the per-store totals to zero on each store header, so a store with no nodes records 0. Converting the version gauge to an observable gauge would also work, but it leaves read-only handles publishing the other migration counters, so they are cut off at the router instead.

No consensus, state, or wire-format impact: only metric emission changes, and the option is variadic, so existing BuildRouter callers are unchanged. A store absent from an export entirely (rather than exported empty) still keeps its last value; that does not occur for evm after the migration. Affected nodes show correct values after deploy, once they restart and export their next snapshot. TestLoadVersionReadOnlyDoesNotReportMigrationVersion and TestSnapshotReportsZeroKeysForEmptyStore each fail without their half of the fix; the migration, composite and rootmulti suites pass.

…ty stores in snapshot key totals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

…call site Co-authored-by: Cursor <cursoragent@cursor.com>

chatgpt-codex-connector · 2026-10-02T19:53:41Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-02T19:57:30.146490Z	`ebced72`	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

cursor · 2026-10-02T19:53:59Z

PR Summary

Low Risk
Observability-only changes to OTel gauge emission and snapshot export counters; no consensus, state, or API behavior changes.

Overview
Fixes stale OpenTelemetry gauges on nodes that export state-sync snapshots after FlatKV/EVM migration has finished.

Migration metrics: BuildRouter gains optional RouterOptions; WithoutTelemetry() wires migration managers to in-process-only metrics instead of process-wide OTel. Derived composite stores (LoadVersionReadOnly, Copy) pass this option so snapshot export no longer overwrites seidb_migration_version with the mid-migration value seen at the export height.

Snapshot key counts: When rootmulti Snapshot emits each store section header, it resets per-store key/value/total counters to zero so an empty memIAVL evm tree still records iavl_total_num_keys as 0 instead of retaining the last pre-migration reading.

Adds regression tests for both behaviors. No consensus, state, or wire-format changes—metric emission only.

^{Reviewed by Cursor Bugbot for commit 4e2f62b. Bugbot is set up for automated code reviews on this repo. Configure here.}

github-actions · 2026-10-02T19:54:41Z

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Oct 3, 2026, 1:54 AM

seidroid

Derived composite stores (read-only views and copies) now build their MigrationManager with local-only metrics, so they stop overwriting the live store's process-wide migration gauges, and rootmulti Snapshot now records zero for a store exported with no nodes. I found nothing blocking: only derived stores take the WithoutTelemetry option, store headers in the export are unique per stream so resetting them loses no counts, and codex's reading that found nothing contributed no findings and matches mine (I could not run the new tests because the sandbox has no Go toolchain).

Pre-existing

Already true on the base branch, not introduced here.

suggestion — In rootmulti Snapshot, a store that is left out of the export entirely still keeps its last iavl_total_* values, because only stores seen in the stream get recorded. CompositeCommitStore.Exporter sets includeMemiavl to false once MigrateBank completes, so after that every memiavl store's key and byte gauges stay frozen at their last values. This is the same stale-dashboard symptom, just later in the migration, so it is worth fixing in a follow-up.

_{seidroid review · decision approve · session 32fb13770ade4a2caab92420ca61aec1 · turn resp_claude_b57292621bf2478dae1f2cd5518ed279 · item f054e084a6ae508a917c46aa98873441}

Findings: 0 blocking | 0 non-blocking | 0 posted inline | 1 pre-existing

codecov · 2026-10-02T19:58:43Z

Codecov Report

❌ Patch coverage is 85.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.74%. Comparing base (84852aa) to head (4e2f62b).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-db/state_db/sc/migration/router_builder.go	76.92%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4436      +/-   ##
==========================================
+ Coverage   56.46%   56.74%   +0.28%     
==========================================
  Files        2123     2126       +3     
  Lines      166573   166836     +263     
==========================================
+ Hits        94050    94675     +625     
+ Misses      72518    72156     -362     
  Partials        5        5

Flag	Coverage Δ
sei-chain	`54.92% <100.00%> (+0.27%)`	⬆️
sei-db	`74.81% <ø> (ø)`
sei-db-state-db	`78.86% <82.35%> (+0.25%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-cosmos/storev2/rootmulti/store.go	`75.33% <100.00%> (+0.10%)`	⬆️
sei-db/state_db/sc/composite/store.go	`79.16% <100.00%> (+0.09%)`	⬆️
sei-db/state_db/sc/migration/router_builder.go	`74.64% <76.92%> (-0.45%)`	⬇️

... and 37 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Co-authored-by: Cursor <cursoragent@cursor.com>

seidroid · 2026-10-04T18:40:26Z

Successfully created backport PR for release/v6.7:

Backport release/v6.7: fix(seidb): fix stale FlatKV migration gauges on snapshotting nodes #4442

…s on snapshotting nodes (#4442) Backport of #4436 to `release/v6.7`. --------- Co-authored-by: yirenz <blindchaser@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: blindchaser <zengyiren0@gmail.com>

devin-ai-integration Bot and others added 2 commits October 2, 2026 14:15

Keep read-only migration handles off the version gauge and report emp…

248ce88

…ty stores in snapshot key totals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Describe WithoutTelemetry by what it does and move the reason to its …

ebced72

…call site Co-authored-by: Cursor <cursoragent@cursor.com>

seidroid Bot approved these changes Oct 2, 2026

View reviewed changes

blindchaser changed the title ~~fix(seidb): stop stale FlatKV migration gauges on snapshotting nodes~~ fix(seidb): fix stale FlatKV migration gauges on snapshotting nodes Oct 2, 2026

blindchaser added backport release/v6.7 Backport to release v6.7 non-app-hash-breaking labels Oct 2, 2026

blindchaser requested a review from alexander-sei October 2, 2026 20:15

Merge branch 'main' into devin/1790957408-flatkv-migration-gauges

4e2f62b

Co-authored-by: Cursor <cursoragent@cursor.com>

alexander-sei approved these changes Oct 3, 2026

View reviewed changes

blindchaser added this pull request to the merge queue Oct 4, 2026

Merged via the queue into main with commit 82c0cda Oct 4, 2026
60 checks passed

blindchaser deleted the devin/1790957408-flatkv-migration-gauges branch October 4, 2026 18:39

seidroid Bot mentioned this pull request Oct 4, 2026

Backport release/v6.7: fix(seidb): fix stale FlatKV migration gauges on snapshotting nodes #4442

Merged

Navigation Menu

Repository navigation

fix(seidb): fix stale FlatKV migration gauges on snapshotting nodes - #4436

fix(seidb): fix stale FlatKV migration gauges on snapshotting nodes#4436
blindchaser merged 3 commits into
mainfrom
devin/1790957408-flatkv-migration-gauges

blindchaser commented Oct 2, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

cursor Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

seidroid Bot left a comment •

edited

Loading

Uh oh!

codecov Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

seidroid Bot commented Oct 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

blindchaser commented Oct 2, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

cursor Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seidroid Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Pre-existing

Uh oh!

codecov Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

seidroid Bot commented Oct 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chatgpt-codex-connector Bot commented Oct 2, 2026 •

edited

Loading

cursor Bot commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading

seidroid Bot left a comment •

edited

Loading

codecov Bot commented Oct 2, 2026 •

edited

Loading